{ "cells": [ { "cell_type": "markdown", "id": "06b88c82-692a-41ab-ac38-1b368407e234", "metadata": {}, "source": [ "# Using ``dask.distributed`` as Parallel Pool Example\n", "\n", "This example shows how you can use any parallel pool-executor like object to run combos - specifally the [dask.distributed.Client](https://distributed.readthedocs.io/en/latest/client.html)." ] }, { "cell_type": "code", "execution_count": 1, "id": "8931d06b-ea8e-4cd1-88fe-b1db58655de9", "metadata": {}, "outputs": [], "source": [ "%config InlineBackend.figure_formats = ['svg']\n", "import numpy as np\n", "from dask.distributed import Client\n", "\n", "import xyzpy as xyz" ] }, { "cell_type": "markdown", "id": "aa962487-523b-4289-b89c-8ecd3af544b8", "metadata": {}, "source": [ "First we instantiate the client:" ] }, { "cell_type": "code", "execution_count": 2, "id": "a7a53fca-4e31-442d-a666-993d4eb434e3", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "
\n", "
\n", "

Client

\n", "

Client-a5c01656-2b14-11f1-b0ab-028aa0870bc6

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
Connection method: Cluster objectCluster type: distributed.LocalCluster
\n", " Dashboard: http://127.0.0.1:8787/status\n", "
\n", "\n", " \n", "\n", " \n", "
\n", "

Cluster Info

\n", "
\n", "
\n", "
\n", "
\n", "

LocalCluster

\n", "

bfad87d4

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "\n", " \n", "
\n", " Dashboard: http://127.0.0.1:8787/status\n", " \n", " Workers: 5\n", "
\n", " Total threads: 10\n", " \n", " Total memory: 32.00 GiB\n", "
Status: runningUsing processes: True
\n", "\n", "
\n", " \n", "

Scheduler Info

\n", " \n", "\n", "
\n", "
\n", "
\n", "
\n", "

Scheduler

\n", "

Scheduler-bbb787eb-7944-4f51-8235-891aee28fdf3

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", " Comm: tcp://127.0.0.1:55507\n", " \n", " Workers: 0 \n", "
\n", " Dashboard: http://127.0.0.1:8787/status\n", " \n", " Total threads: 0\n", "
\n", " Started: Just now\n", " \n", " Total memory: 0 B\n", "
\n", "
\n", "
\n", "\n", "
\n", " \n", "

Workers

\n", " \n", "\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 0

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://127.0.0.1:55523\n", " \n", " Total threads: 2\n", "
\n", " Dashboard: http://127.0.0.1:55524/status\n", " \n", " Memory: 6.40 GiB\n", "
\n", " Nanny: tcp://127.0.0.1:55510\n", "
\n", " Local directory: /var/folders/qg/9jgh3y7x2352yjf6l63gzhyc0000gn/T/dask-scratch-space/worker-5s22_cq1\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 1

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://127.0.0.1:55520\n", " \n", " Total threads: 2\n", "
\n", " Dashboard: http://127.0.0.1:55521/status\n", " \n", " Memory: 6.40 GiB\n", "
\n", " Nanny: tcp://127.0.0.1:55512\n", "
\n", " Local directory: /var/folders/qg/9jgh3y7x2352yjf6l63gzhyc0000gn/T/dask-scratch-space/worker-d2l7t2jh\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 2

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://127.0.0.1:55526\n", " \n", " Total threads: 2\n", "
\n", " Dashboard: http://127.0.0.1:55527/status\n", " \n", " Memory: 6.40 GiB\n", "
\n", " Nanny: tcp://127.0.0.1:55514\n", "
\n", " Local directory: /var/folders/qg/9jgh3y7x2352yjf6l63gzhyc0000gn/T/dask-scratch-space/worker-d6gshrhb\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 3

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://127.0.0.1:55529\n", " \n", " Total threads: 2\n", "
\n", " Dashboard: http://127.0.0.1:55530/status\n", " \n", " Memory: 6.40 GiB\n", "
\n", " Nanny: tcp://127.0.0.1:55516\n", "
\n", " Local directory: /var/folders/qg/9jgh3y7x2352yjf6l63gzhyc0000gn/T/dask-scratch-space/worker-qcvsx1vj\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 4

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://127.0.0.1:55532\n", " \n", " Total threads: 2\n", "
\n", " Dashboard: http://127.0.0.1:55533/status\n", " \n", " Memory: 6.40 GiB\n", "
\n", " Nanny: tcp://127.0.0.1:55518\n", "
\n", " Local directory: /var/folders/qg/9jgh3y7x2352yjf6l63gzhyc0000gn/T/dask-scratch-space/worker-hzoz0v4z\n", "
\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
\n", "\n", "
\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
" ], "text/plain": [ "" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "address = None # put in an actual address like '127.0.0.1:8786' if you have a scheduler running\n", "c = Client(address=address)\n", "c" ] }, { "cell_type": "markdown", "id": "25162aca-1b74-43d6-8529-046d4f3c7a39", "metadata": {}, "source": [ "The ``Client`` instance can act like a normal parallel pool since it has a submit method and returns futures:" ] }, { "cell_type": "code", "execution_count": 3, "id": "06d4e3a8-fd6c-4440-9aee-e981c4a36047", "metadata": {}, "outputs": [ { "data": { "text/plain": [ ">" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "c.submit" ] }, { "cell_type": "markdown", "id": "917dcfde-b6a0-4dbf-a59d-2dcc92a4bfa0", "metadata": {}, "source": [ "Let's define our test function and ``combos``:" ] }, { "cell_type": "code", "execution_count": 4, "id": "6036ba91-2479-437b-8536-104aadb222cc", "metadata": {}, "outputs": [], "source": [ "def rad(x, y, n):\n", " r = (x**n + y**n) ** (1 / n)\n", " if r == 0.0:\n", " return 1\n", " return np.sin(r) / r\n", "\n", "\n", "combos = {\n", " \"x\": np.linspace(-15, 15, 51),\n", " \"y\": np.linspace(-15, 15, 51),\n", " \"n\": [2, 4, 6],\n", "}\n", "\n", "r = xyz.Runner(rad, var_names=\"sinc2d\")" ] }, { "cell_type": "markdown", "id": "d2458a71-6fee-47e5-a52d-dc89d3fa892f", "metadata": {}, "source": [ "Now we can run (or ``harvest``, or ``grow_missing``) our combos, supplying the ``Client`` instance to the ``executor=`` keyword:" ] }, { "cell_type": "code", "execution_count": 5, "id": "e54742f4-ccc7-4a65-b208-078fa6622f69", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "100%|##########| 7803/7803 [00:10<00:00, 719.90it/s]\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset> Size: 63kB\n",
       "Dimensions:  (x: 51, y: 51, n: 3)\n",
       "Coordinates:\n",
       "  * x        (x) float64 408B -15.0 -14.4 -13.8 -13.2 ... 13.2 13.8 14.4 15.0\n",
       "  * y        (y) float64 408B -15.0 -14.4 -13.8 -13.2 ... 13.2 13.8 14.4 15.0\n",
       "  * n        (n) int64 24B 2 4 6\n",
       "Data variables:\n",
       "    sinc2d   (x, y, n) float64 62kB 0.03308 -0.04752 ... -0.04752 -0.05369
" ], "text/plain": [ " Size: 63kB\n", "Dimensions: (x: 51, y: 51, n: 3)\n", "Coordinates:\n", " * x (x) float64 408B -15.0 -14.4 -13.8 -13.2 ... 13.2 13.8 14.4 15.0\n", " * y (y) float64 408B -15.0 -14.4 -13.8 -13.2 ... 13.2 13.8 14.4 15.0\n", " * n (n) int64 24B 2 4 6\n", "Data variables:\n", " sinc2d (x, y, n) float64 62kB 0.03308 -0.04752 ... -0.04752 -0.05369" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "r.run_combos(combos, executor=c)" ] }, { "cell_type": "markdown", "id": "457e5d14-2886-42c2-a95e-4829f753f2cd", "metadata": {}, "source": [ "That should take enough time to check out client status including the tasks being processed on for example: http://127.0.0.1:8787/tasks.\n", "\n", "Finally, visualize the output:" ] }, { "cell_type": "code", "execution_count": 6, "id": "608371c4-2aad-4235-adb6-23d7f3074999", "metadata": {}, "outputs": [ { "data": { "image/svg+xml": [ "" ], "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "data": { "text/plain": [ "(
,\n", " array([[, ,\n", " ]], dtype=object))" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "r.last_ds.xyz.infiniplot(\"x\", \"y\", z=\"sinc2d\", col=\"n\", palette=\"turbo\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3" } }, "nbformat": 4, "nbformat_minor": 4 }